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trao^detAILED DESCRIPTION OF THE INVENTION 

25. The descriptions that follow are provided so as to enable any person 
skilled in the art to make and use the invention, and sets forth some 
modes presently contemplated by the inventors of carrying out their 
invention. Various modifications, however, will remain readily apparent to 
those skilled in the art, since the generic principles of the present invention 
have been defined herein. 



26.The Camera System means, illustrated in Figure 1., of this invention 
incorporates the use of, but not limited to, sensors such as a surveillance 
video camera means with incorporated local controller, 101 , with forward, 
102, and backward listening audio detection capability, 103, and video 
detection viewing capability, 104, functions and the camera system could 
but not necessarily have, full pan, tilt and zoom computer controlled 
motion for monitoring of a given scene, situation, place, thing, persons or 
environment. The Camera System means can provide the camera 
means, 101, with communications means for communications means of 
hardwired, 105, or wirelessly connected, 106, to a central processing and 
security systems/ facility, which may be located at the facility where the 
surveillance is occurring or at a remote location, either of which, with 
incorporated central controller, 107, to storage systems, connecting to 
databases, connecting to fuzzy logic processors, and/ or connecting to 
mobile portable cellular or FM radio phones, 108, or connecting to PC 
facilities and connecting to personal data assistants (PDA's such as 
"Palm" or "Pocket PC" and the like), 109. Intelligent analysis and fuzzy 
logic functions means may be integrated into the camera means or any of 
the so connected or connectable facilities forming the complete Camera 
System means of figure 5. 



27. In a preferred embodiment, camera system means illustrated in Figure 1., 
of this invention could be observing a person or persons and their activity, 
110, with a plurality of sensors such as but not limited to audio and video 
so that the intelligent analysis means can interpret those person, persons 
and/ or activities for potential threat and can record the speech and facial 
observations for identification using but not limited to comparisons to 
databases of audio, visual and speech and text information to which the 
facilities are connected via the Internet WEB, 1 1 1 , or by hardwired land, 
112, or telephonic or wireless links, 108 and 109. 

28. A unique aspect of the Camera System means of this invention illustrated 
in Figure 1., is its ability to learn from the data collected from these 
observations and from data in and collected for the comparison databases 
means. These database means and facilities, whether incorporated into 
the camera means or located elsewhere, can include local and remote 
databases including but not limited to: the Multi-Media, 113, such as print 
including news papers, radio and TV; the Government, 114, such as 
criminal activity/ conviction or drivers license identification or terrorist 
activity; and the Associated data systems, 115, such as medical/ mental 
health, or incarceration, or education, and the like. Health information, 
accessed by the Camera System means, in particular could be critical in 
understanding the actions of persons to recognized the differences 
between drunkenness, heart attack, diabetic coma, epileptic seizure and 
the like. These databases and their information as accessed by the 
Camera System means can become as part of the Camera System 
means and can be linked via the WEB, hardwired, telephony or wireless 
means. Learning by the Camera System means by comparison of the 
observations it makes with video and audio detection to the information 
available in these databases, can result from the Camera System means 
making a plurality of analyses, conducted by the Intelligent Camera 
System means in Figure 5., 503, such as but not limited to the Intelligent 



Camera System use of fuzzy logic processor, 501, and resulting 
comparisons with information stored in the general facility databases, 502, 
including but not limited to previous visual data such as faces of persons, 
and/ or audio data such as speech, and/ or text data such as key words or 
phrases. Such comparisons from which the system using fuzzy logic 
matches those aspects of the current observations, the system can 
proceed to learn that the person being observed has, for example, used 
the word "Bomb" but because, for example, the person's actions in 
reference to a database of actions can not be considered a threat and, for 
example, the person's face can not be recognized in reference to a 
database of faces, hence the fuzzy logic processor, 501, of the Camera 
System means in this example, from such learning, decides to save the 
new face and speech and key word "bomb" for future comparison. The 
Camera System means so learns that this person currently under 
surveillance observation is a person for the Camera System means to 
"watch out for" in the future and at the same time it decides to continue 
current observations of this person, in case further data matching to the 
databases indicates a true or potential threat exists or could occur. 

29. A plurality of threats or threatening conditions can be detected and 
responded to by this Camera System means including but not limited to 
terrorist threats, personal threats physical or otherwise, telephony 
harassment, unauthorized entry or exit, dishonesty and fraudulence such 
as misuse of credit cards or smart cards or proximity identification systems 
and the like threats. To all of these potential conditions, actions or people, 
it is the observation, the data collection, the database structures and 
storing of information for comparisons and the fuzzy logic analyses and 
interpretation from any, some, or all data from the databases or from 
observations of persons, places and things and the resulting recognition of 
their meaning and possibility of threat that this invention reveals. 
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30. The Intelligent Camera system Means of Figure 5., that incorporates a 
fuzzy logic processor, 501, means of this Camera System means will 
include a plurality of computer analyses techniques and technologies, 
software, firmware and hardware methods and designs including but not 
limited to recording and storage and retrieval of data, video pattern 
recognition, facial recognition, body action recognition, stress analysis of 
facial appearance and movement, stress analysis of body appearance and 
movement, emotional condition stress analysis from facial and/ or speech 
and/ or body action, surrounding environment condition assessment, voice 
stress analysis, voice recognition, voice speech recognition to text, lip 
reading recognition of speech and conversion to text, deep extraction of 
information and meaning from text or multi-media information, and the like. 

31. Many of these techniques and technologies have been noted in the 
background to this invention, but what is unique in this invention is that we 
reveal a Camera System means for the combination of these into one 
integrated system: a "Voice, Lip-reading, Face and Emotion Stress 
Analysis, Fuzzy Logic Intelligent Camera System" for intelligent analysis 
and automated learning and decision analysis for the detection and 
understanding of a threat or potential threat by a person or persons or 
animals or objects, or by their actions, or by their appearance or by their 
speech or any combination of these. 

32. An illustrated example could be a threat to people and/ or property. 
Referring to Figure 2, we use the application of this invention to the sports 
arena environment and more specifically addressing the security problem 
of sports fans posing a potential threat to the sports athletes, other fans 
and persons at the arena or the arena itself, which application of our 
invention we refer to as a process of scanning the sports fans. This could 
include but not limited to throwing objects at athletes or fans or others at 
the arena; fans going onto the athletic field and interfering with the 



athletes or activities on the field; fans or persons being loud and/ or 
abusive; fans or persons being intoxicated; fans or persons being a threat 
to their own well-being and the like. Figure 2, shows the Toronto, Ontario, 
"SkyDome" as the example arena in which we have indicated six 
surveillance audio-video camera means, 201 , (1 to 6) located at field level 
and two more, 202, (7 & 8) at the 200 level. Also indicated on the 
concourse level, 203, are 21 usher/ security personnel (a) through (u), 
located near the entrance to each aisle of the field level seats as well as 
locations, 204, of 2 beer purchasing counters. An optional additional 15 
cameras, 205, G-1 through G15 could be positioned at each of the 
entrance gates to the arena. 

33. In this example of a preferred embodiment of the Camera System means, 
illustrated in Figure 2., the six camera means, 201, with full pan, tilt and 
zoom functions have the software and hardware means to continuously 
scan the fans in the stands and the system has the analysis and database 
comparison information means in order to recognize and detect sounds, 
speech, actions or activities of the fans that may indicate a threat or 
threats such as noted. The camera means having forward listening and 
backward listening microphone means, 102 and 103, and through the 
Intelligent Camera System means fuzzy logic capability illustrated in 
Figure 5., 501, can access and utilize the Camera Means Information 
logic indicated in Figure 6, for camera control means, 607, with analyses 
to effect the camera means to point in the direction from which the 
microphones detected sound that the fuzzy logic decided requires video 
surveillance, can be so pointed by the Camera System means. The 
camera means so directed views the area from which the sound was or 
may still be emanating and further fuzzy logic analyzes the images and 
the sounds being observed and with the analysis means, compares them, 
through the communication linkages, illustrated in Figure 1., 111 or 112, to 
information in the databases from which the camera system means fuzzy 



logic can analyze the observed and database accessed informantion and 
can assess possible threat. 

34. Many possible comparisons by the Camera System means fuzzy logic 
could occur, such as but not limited to: a person being observed, 110, by 
the video camera of the Camera System means of Figure 1, and the 
analyses of that video data by the Intelligent Camera System means of 
Figure 5. using the fuzzy logic of the Camera Means Information of Figure 
6. , 601, is matched by the Camera Means Information analysis, 602, to 
the image of a person guilty of assault recorded in one of the databases, 
603; or with similar steps of logic for the case of audio data observations 
of the person, the language of a person being observed is, or could be 
interpreted as emotionally angry or abusive or intoxicated or a 
combination of these; or with the video data analyses, the actions of a 
person being observed indicate likelihood of the person attempting to 
throw an object at an athlete on the athletic field; or the actions of a 
person being observed indicate the likelihood of the person attempting to 
enter the athletic field or such like. If and when Camera System means 
fuzzy logic incorporated in the fuzzy logic processor illustrated in Figure 5., 
501, accessing the logic of the Camera Means Information of Figure 6., 
finds a match that is a threat, 604, then at any facility employing the 
Camera System Mean it can inform the security facility, 605, at any such 
facility in this example at the arena such as at the Central Facility 
indicated in Figure 1., 107, or the security personnel via wireless 
communications, 108 or 109, for appropriate actions, that the Camera 
Means Information fuzzy logic indicated in Figure 6., decides, 606, to be 
taken. 



35. Alcohol abuse and intoxication are well known as a related contributing 
factor to fan-caused disruption and threat at athletic arenas and for such, 
the Camera System means can implement a unique sensor, data 



collection and database logging means. We reveal the new idea of 
tracking tickets of fans in the venue such as a sports arena and in so 
tracking the tickets we reveal a method for also tracking the purchase of 
alcohol. It is also contemplated in this invention, that a new "Visual 
Response Measure" can be established and used to identify the that a 
person is drunk, intoxicated or impaired by alcohol or drugs and that 
standard sets of levels of such intoxication or impairment can be identified 
and used to classify a person's level of intoxication or impairment. 

36. Tracking of tickets can be done using the well known magnetic strip 
recording/ data storage means indicated in Figure 3., 301, used with credit 
or debit cards, or parking lot entrance/ exit tickets and the like, or smart 
cards, or proximity cards, and incorporating one or more of these 
technologies into the physical tickets to the arena, 302. Recorded on this 
recording/ data storage means could be information such as but not 
limited to the seat or location allowed fan access, the date and 
performance being attended, the cost of the ticket, sales location, method 
of payment for the ticket and the like. In the case of seasons ticket 
holders, the seat location information could be preloaded into the 
databases of the Camera System means ahead of actual attendance at 
the arena; the holder's identification information could be stored for future 
reference by the Camera System means as well. The recording means on 
the ticket could also be a printed means, such as bar code or magnetic 
printing and the like, for which the reading/ writing means would be 
designed and implemented also to read and write these. 

37. Usually the fan retains a portion of the ticket as proof of paid entry and for 
display to security people to access the fan's seat or location during his 
attendance at the arena. The ticket recording/ data storage means could 
be located only on the ticket portion retained by the fan, 302, or could also 
could be located on the portion retained by the arena. At the entry gates 



to the arena the ticket reading logic as illustrated in Figure 8., 809, located 
at said gates as illustrated in Figure 2, 205, using the reading means 
illustrated in Figure 3., 303 and fuzzy logic of the Camera Means 
Information accessing and analyzing said reading as illustrated in Figure 
6, 608, could be located to read, 609, and store with the central facility, 
304 noted in Figure 3. and the reader data analysis fuzzy logic 
interpretation of said reading, illustrated in logic of Figure 8., 810, 
communicated via using wired or wireless communications, illustrated in 
Figure 3., 305, the data recorded on the ticket as indication of entrance, 
and as communicated to the Camera Means Information system logic 
indicated in Figure 6., 608, 609, 610 and 61 1 , of a fan holding that ticket. 
Either the fan or security personnel, operating the reader system indicated 
in Figure 3., the security personnel or the fan, 306, could place the ticket 
into the reading means and retrieve it following its being read and the fan 
could proceed to enter the arena taking his portion of the ticket with him. 
If the ticket recording/ data storage means is located just on the ticket 
portion to be retained by the fan, then this portion or the whole ticket 
would need to be placed in the reading means and the fan would have to 
wait for the reading task to be completed before entering the arena with 
his portion of the ticket removed from that portion retained by the arena. It 
is also possible that the reading means, indicated in Figure 3., 303, could 
incorporate a writing means, 307, for the Camera System means to 
automatically write on the ticket recording/ data storage means, 308, 
additional information such as but not limited to gate and time of entry. It 
would be possible also to include a numeric input key pad on the reading 
means, 303, to enter information, which subsequently is communicated to 
the Camera Means Information system of Figure 6., 61 1 , manually for the 
Camera System means to write the additional information onto the 
recording/ data storage means. 



38. It is not necessary for the arena to retain a ticket portion or even for the 
ticket to be separable into portions; however, for the following ticket 
tracking use in monitoring and controlling alcohol use, it is necessary that 
the ticket recording means at least be located on the ticket or ticket portion 
retained by the fan. If a separable ticket is used, the fan or the arena 
personnel at the entrance gate, could have the ticket information read into 
the reader of Figure 3., 303, communicated to the Camera Means 
Information fuzzy logic system of Figure 7., 709, could separate the ticket 
portions at the entrance gate and only that portion retained by the arena 
need be read, and communicated to the fuzzy logic system of Figure 7., 
710, thereby allowing the fan to proceed into the arena with his portion of 
the ticket and without waiting for the reading means to complete its task, 
thereby saving time and speeding up entrance of fans. However, if it is 
desired to have the writing means add data to the recording/ data storage 
means on the ticket portion retained by the fan, it would be necessary to 
have that fan's portion placed in the reading and writing means and to be 
written upon. If the fan does the placing of the ticket or ticket portion into 
the reading means and the gate means could be automated to allow entry 
if the ticket is accepted by reading means, it is possible that arena entry 
could be "self serve" and the need for security personnel on every gate 
could be reduced or eliminated resulting in significant cost savings to the 
arena. This ticket reading method also clearly provides significant 
statistical data information gathering function, of the fuzzy logic analysis 
system indicated in Figure 7., 711, capability at the central facility relating 
to arena management and operations. For season's ticket holders, their 
identification information as well as seat location could be automatically 
entered into the Camera System databases at ticket purchase thereby 
enriching the data information. 

39. If ticket tracking information is combined with the camera means, and 
ticket reading function located a the area entry gates such as Gate 9 



indicated in Figure 2., 205, and, the Camera System Means indicated in 
Figure 1., 101 and in Figure 3., 309, observing the fan upon entrance, 
then the fuzzy logic system indicated in the Camera Means Information 
system of Figure 7., 712, together they could provide a major source of 
information, analyzed by the Ticket Card Reader Means Information 
system indicated in Figure 8., 81 1 , for recording and storing these data 
from the card reader using the fuzzy logic of the Ticket Card Reader 
Means Information system, 812, into the security fuzzy logic analysis, 813, 
database of the Camera System means as a function of the Ticket Card 
Reader Means Information system fuzzy logic, 814. If the ticket reading 
means, 303, illustrated in Figure 3., also has a writing means, 307, as 
proposed above, data regarding the camera means observations could 
also be added to the information recorded by the Camera Measns 
Information system illustrated in Figure 7., 711, on the ticket recording/ 
data means, printing on the fan's ticket indicated in Figure 3., 308, such as 
but not limited to identification for the images recorded by the camera 
means viewing the fan at the time of entering the arena via the gate being 
viewed, or writing, and recording these audio and video and ticket read 
data into a database for future reference, 713, including video data 
recording an actual image of the fan so recorded on the recording/ data 
storage means, or the like. 

40. With ticket tracking identification information such as the fan's seat 
location and the date and the performance at which the fan is attending 
recorded on the ticket recording/ data storage means, a reading means, 
as illustrated in Figure 3., 303, and if the reading and writing means is 
equipped with a visual display means, 310, and such reading and writing 
means is also installed, for example, at all alcohol vending areas, and the 
said fan's ticket is also so read at these vending areas, then the fuzzy 
logic system of the Ticket/Card Reading Means Information system 
indicated in Figure 8., 801, like the beer counters illustrated in the arena 



shown in Figure 2., 204, it could be arranged such that the ticket of the 
alcohol purchaser must first be read, and that information read from the 
ticket communicated to the Ticket/ Card Reader Means Information 
system indicated in Figure 8., 802, by the reading means and information 
so read and stored by the ;fuzzy logic of the Ticket/ Card Reader Means 
Information system, 803, on one of the databases of the Camera System 
means before each and every purchase. The fuzzy logic of the Camera 
System means at the central facilities being part of the Camera Means 
Information system illustrated in Figure 6., 612, would then add this 
current alcohol sale to the database information on this fan's alcohol 
purchases by the fuzzy logic of the Ticket/ Card Reader Means 
Information system of Figure 8., 804, such that the fuzzy logic system 
could request to the total of all previous such sales so recorded that day 
as accounted to this said ticket and only if that total so computed , 805, is 
less than a preset amount comparison made by the fuzzy logic of the 
Camera Means Information system indicated in Figure 6., 613, will the 
Camera System means display on the visual display unit located at the 
said alcohol sales location illustrated in Figure 8., 806, indicate approval 
for alcohol purchase if the total to that time is less than the preset limit and 
before the purchase is allowed to proceed by the bar tender serving the 
fan, 807, and if said limit is met or exceeded, the display would indicate 
via the fuzzy logic of the Camera Means Information system illustrated in 
Figure 6., 614, no sale is to be allowed to that ticket holder and would be 
so displayed on the ticket reading and display system illustrated in Figure 
3., 31 1, such as "no sale" and/or a light or lights incorporated on the said 
display system, 312, such as a green illumination indicating "sale 
approved" and a red illumination indicating "no sale" or the like as 
analyzed and decided by the fuzzy logic of the Ticket/ Card Reader Means 
Information system illustrated in Figure 8., 808. This description is 
intended to be for the purchase of alcoholic beverages and display means 
could also include an audible means for indication to the seller, or also 



could include a loud speaker means incorporated into the ticket/ card 
reading system indicated in Figure 3., 313, to simulate the equivalent of 
the bar tender telling the fan patron (or patrons) he is "cut off' from all 
further drinks. This ticket tracking process, although described here for 
alcohol related purchases, is not limited to such and could apply to any 
and all activities of the fan while in the arena. The ticket reading means 
indicated in Figure 3., could also include a writing means either 
electronically recording on the storage means of the ticket or a printing 
means to print a visible mark such as "Void", illustrated in Figure 3., 314, 
indicating no further alcoholic sales to anyone displaying that ticket. 
Policing drinking at athletic events with ticket tracking processes could 
significantly reduce fan rowdiness, drunkenness and threats, and refusing 
to serve fans is one effective way to minimize this threat. 

41. The camera means noted earlier in the SkyDome arena at the 200 level, 
illustrated in Figure 2., 202 and the video camera system illustrated in 
Figure 1., 101, communicating said video surveillance of the said fan to 
the Camera Means Information system fuzzy logic analyses illustrated in 
Figure 7., 701, camera means, Camera Means Information system logic of 
Figure 7. & Ticket/ Card Reader Means Information system reading of the 
fan's ticket information for analyses indicated in Figure 8., equipped with a 
zoom means sufficient to view any seat in the arena, could be combined 
with the 100 level camera means could communicate said video data 
observation to the central Facility Camera Means Information system logic 
and analyses processes indicated in Figure 7., 702, such that if the 100 
level camera means indicated to the Camera System means that a 
security risk or threat exists, then the fuzzy logic analyses of the Camera 
Means Information system 703, of the Camera System means could 
direct one or more of the 200 level camera means resulting from the threat 
analyses of these video data, to then using the fuzzy logic, 704, of the 
Camera Means Information system could direct the said level 200 camera 



means system to zoom in to view the area indicated to provide another 
view of the scene or vice versa. This functionality could be combined with 
the ticket tracking function and the 200 level camera means fuzzy logic of 
the Camera Means Information system direct the said cameras so that 
they could zoom as determined by said fuzzy logic, 705, in to view on the 
very seat for which that ticket was purchased. Additionally, these 200 
level camera means can provide discreet non-intrusive observation of the 
entire arena to consider if any threat has been detected at any location the 
cameras view in the entire arena implementing the fuzzy logic indicated in 
Figure 7., 706. However, clearly every arena will have its own particular 
construction and layout and the use of "200 level" or "100 level" camera 
means is only illustrative of placement of camera means for more distant 
and for more close up viewing and the numbers of camera means and 
sensors and their locations would be selected and arranged appropriate to 
the arena in which they are used. 

42. The Camera System means of this invention is intended as a tool to assist 
security personnel and systems to automatically detect potential threats as 
determined by the fuzzy logic of the Camera Means Information system 
illustrated in Figure 7., 707, and to report these to the security systems 
and personnel, 708, as appropriate. The actions of persons resulting from 
drunkenness include but are not limited to difficulty in walking due to 
drunken condition as well as aggressive behavior such as loud talking and 
yelling and often accompanied with swearing, and are just some of the 
reactions that the Camera System means can be programmed to detect 
and that it can learn to observe. 

43. The sports arena is just one of a plurality of applications to which the 
Camera System means revealed in this invention that anyone skilled in 
the art of security systems and their implementation can immediately 
recognize and understand by this invention. Another example of a 



preferred embodiment is use in border security and security of regions, 
areas, compounds and the like. Figure 4 illustrates an application to a 
USA border crossing in which invalid entry, smuggling, terrorist activity or 
movement are just some of the threats to which the Camera System 
means of this invention could provide a tool to assist security personnel, 
border guards and security systems. 

44. Camera means including audio and video observations and recordings are 
illustrated in Figure 4, as gantry mounted units, 401 , 402, and as a post 
mounted unit, 403, which are arranged to observe the actions, speech, 
emotions, and the like of the occupants of cars approaching and at the 
guard kiosk while the Camera System means automatically analyzes the 
data recorded. As described earlier the Camera System means searches 
its databases and central facilities including stress analysis, lie detection, 
emotional stress, deep text extraction and the like to determine if a threat 
or potential threat or criminal action or terrorist or other undesirable is 
present. If such a threat or potential is detected the Camera System 
means immediately informs the security border guard and the security 
systems and personnel for appropriate actions to be taken. 

45. In this illustration, camera means unit, 401, is observing the vehicle at the 
guard kiosk, and is viewing the occupants recording video images and 
sounds and speech of the occupants and the guard. Camera means, 402, 
is recording the similar information observing the occupants of the next 
vehicle waiting to approach the security border guard; however, this 
camera means will attempt to observe the actions of the occupants and 
their "private" discussions and actions prior to addressing the guard. 
Often discussion between occupants may be very revealing of the 
intentions to carry out threatening or illegal actions. Viewing the next 
vehicle waiting by a camera means at some distance from the vehicle can 
also permit identification of the vehicle type and license plate, and camera 



means, 402, could also be located to view the rear of the waiting vehicle 
and the rear license plate. Camera means, 403, could be located very 
close to the waiting vehicle to more closely view the occupants and if 
possible hear their conversation, and if not audibly hearing their speech, 
then possibly using the Lip Reading observations and analyses revealed 
in this invention to provide the Camera System means with the data 
needed for recognition of text, permitting deep extraction of words, 
phrases and meanings related to threats or potential threats. 
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